Imaging apparatus and image processing method

ABSTRACT

An imaging apparatus and an image processing method capable of increasing correction accuracy of a phase difference detection pixel even in a case where the phase difference detection pixel is densely arranged in an imaging element in order to secure AF performance are provided. An imaging element includes normal pixels of RGB and first and second phase difference pixels of which opening portions are adjacently arranged to face each other in a horizontal direction and in which a G filter is arranged. A pixel value addition unit (64) generates an addition pixel corresponding to a virtual G pixel at a pixel position between the first and second phase difference pixels by adding pixel values of the pair of the first and second phase difference pixels. In a case where the first or second phase difference pixel is set as an in-focus pixel that is an interpolation target, an average value interpolation unit (62) uses the normal pixels surrounding a pixel position of the in-focus pixel and the addition pixel in a case of performing an interpolation operation on a pixel value at the pixel position of the in-focus pixel.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a Continuation of U.S. patent applicationSer. No. 16/562,631, filed Sep. 6, 2019, which is Continuation of PCTInternational Application No. PCT/JP2018/012109 filed on Mar. 26, 2018claiming priority under 35 U.S.C § 119(a) to Japanese Patent ApplicationNo. 2017-068559 filed on Mar. 30, 2017. Each of the above applicationsis hereby expressly incorporated by reference, in its entirety, into thepresent application.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to an imaging apparatus and an imageprocessing method and particularly, to an imaging apparatus comprisingan imaging element including a phase difference detection pixel and animage processing method.

2. Description of the Related Art

Recently, a technology for arranging a pair of phase difference pixelshaving opening portions for pupil separation at different positions as aphase difference detection pixel in an imaging element and performingphase difference AF based on a phase difference between two imagesignals obtained from the pair of phase difference pixels has beenwidely used in order to increase the speed of autofocus (AF)(JP2016-208042A and JP2015-070432A).

The phase difference pixel is formed by covering a part of an opening ofa normal pixel with a light shielding film. For example, in the case ofdesiring to detect the phase difference in a left-right (horizontal)direction on the surface of the imaging element (image sensor), a pixel(first phase difference pixel) obtained by shielding a left side of thenormal pixel from light and a pixel (second phase difference pixel)obtained by shielding a right side of the normal pixel from light areformed. Phase difference detection is performed using pixel valuesobtained from the first phase difference pixel and the second phasedifference pixel. The phase difference pixel has a narrow opening andhas directivity unlike the normal pixel. Thus, the phase differencepixel strongly depends on an incidence ray angle with respect to theimage sensor. The pixel value of the phase difference pixel variesdepending on an image height, an F-number, a defocus amount, and thelike. In the detection of the phase difference, it is necessary toarrange such a pixel at a constant interval on an image surface. Sincethe normal pixel and the phase difference pixel have different pixelcharacteristics, it is necessary to generate a recording image or adisplay image after appropriately correcting the phase difference pixel.

An imaging element disclosed in JP2016-208042A includes a first pixelarray in which a plurality of G pixels detecting light of green (G) arearranged at a predetermined pitch in the horizontal direction (Xdirection), and a second pixel array in which a plurality of B pixelsdetecting light of blue (B) and R pixels detecting light of red (R) arealternately arranged at the predetermined pitch in the horizontaldirection (Y direction), A plurality of first pixel arrays and secondpixel arrays are alternately arranged at the predetermined pitch in theY direction. In addition, the first pixel array and the second pixelarray are arranged to deviate from each other by half of thepredetermined pitch in the Y direction. Furthermore, each side of eachunit pixel of the imaging element is arranged to be inclined at 45degrees with respect to the X direction and the Y direction (FIG. 5 andParagraph 0020 in JP2016-208042A).

SUMMARY OF THE INVENTION

In the imaging element disclosed in JP2016-208042A, the G pixel (virtualG pixel) can be generated at a position between the pair of phasedifference pixels by adding the pixel values of the pair of phasedifference pixels to calculate the double of the pixel value. However,it is also necessary to add the pixel values of two adjacent pixels ofthe same color for the normal pixel (the G pixel, the R pixel, and the Bpixel in which the phase difference pixel is not assigned) other thanthe phase difference pixel. A problem arises in that the resolutions ofthe recording image and the display image is decreased by half.

In addition, the imaging element disclosed in JP2016-208042A has aspecial arrangement such that the horizontal centroid of the added phasedifference pixels deviates from the horizontal centroid of the addednormal pixels. In addition, a color arrangement of virtual R pixels, Gpixels, and B pixels formed after the addition corresponds to the Bayerarrangement which is a square arrangement. Thus, the imaging element isa special imaging element in which each side of each unit pixel of theimaging element is arranged to be inclined at 45 degrees with respect tothe X direction and the Y direction.

JP2015-070432A discloses average value interpolation in which pixelvalues of a plurality of surrounding normal pixels are used ininterpolation of the pixel value of the phase difference pixel. However,the idea of adding the pixel values of the pair of phase differencepixels is not disclosed. In a case where the pixel values of the pair ofphase difference pixels are added (used in the average valueinterpolation), the accuracy of the average value interpolation isdecreased because the pair of phase difference pixels disposed in animaging element disclosed in JP2015-070432A is not adjacent to eachother.

The present invention is conceived in view of such matters. An object ofthe present invention is to provide an imaging apparatus and an imageprocessing method capable of increasing correction accuracy of a phasedifference detection pixel even in a case where the phase differencedetection pixel is densely arranged in an imaging element in order tosecure AF performance.

In order to achieve the above object, an imaging apparatus according toone aspect of the present invention comprises an imaging element inwhich a plurality of phase difference detection pixels and a pluralityof normal pixels are two-dimensionally arranged in a first direction anda second direction orthogonal to the first direction, the phasedifference detection pixel includes a first phase difference pixel and asecond phase difference pixel including opening portions for pupilseparation at different positions in the first direction, and theopening portions of the first phase difference pixel and the secondphase difference pixel are adjacently arranged to face each other, inthe plurality of normal pixels, a first filter corresponding to a firstcolor most contributing to obtaining a brightness signal and a pluralityof second filters respectively corresponding to two or more colors otherthan the first color are arranged in a first periodic color arrangement,and in each of the first phase difference pixel and the second phasedifference pixel, the first filter is arranged, or light in a wavelengthrange wider than a transmission wavelength range of the first filter isincident; a pixel value addition unit that generates an addition pixelvalue at a pixel position between the first phase difference pixel andthe second phase difference pixel by adding pixel values of the firstphase difference pixel and the second phase difference pixel of whichthe opening portions are adjacently arranged to face each other; and afirst interpolation unit that sets the first phase difference pixel orthe second phase difference pixel as an in-focus pixel and generates apixel value at a pixel position of the in-focus pixel by using pixelvalues of pixels surrounding the pixel position of the in-focus pixel,the addition pixel value added by the pixel value addition unit beingused as a pixel value of one pixel of the surrounding pixels.

In a case where the pixel values of the pair of the first phasedifference pixel and the second phase difference pixel of which theopening portions are adjacently arranged to face each other are added, avirtual pixel (pixel including the first filter) can be created betweenthe pair of the first phase difference pixel and the second phasedifference pixel.

According to one aspect of the present invention, a pixel value(addition pixel value) of the virtually created pixel is used in thecase of interpolating a pixel value for imaging at the pixel position ofthe first phase difference pixel or the second phase difference pixel(in-focus pixel) which is a correction target. Thus, interpolationaccuracy of the phase difference detection pixel (in-focus pixel) can beincreased. Accordingly, even in a case where the phase differencedetection pixel is densely arranged in the imaging element in order tosecure AF performance, the interpolation accuracy decreased by the densearrangement can be complemented using the addition pixel value.

It is preferable that the imaging apparatus according to another aspectof the present invention further comprises an addition pixel levelcorrection unit that corrects the addition pixel value by multiplyingthe addition pixel value added by the pixel value addition unit by a setlevel adjustment coefficient, and the first interpolation unit uses theaddition pixel value corrected by the addition pixel level correctionunit.

According to another aspect of the present invention, the addition pixelvalue is corrected by multiplying the addition pixel value obtained byadding the pixel values of the pair of the first phase difference pixeland the second phase difference pixel by the level adjustmentcoefficient. Thus, the addition pixel value after correction can be setto completely match a pixel value of the normal pixel which includes thefirst filter and of which the pixel value may be obtained in a casewhere the normal pixel is present at the same pixel position.

In the imaging apparatus according to still another aspect of thepresent invention, it is preferable that the first interpolation unituses at least one pixel value of the addition pixel value or a pixelvalue of the normal pixel in which the first filter is arranged. Thatis, the first interpolation unit may use only the addition pixel valuein the interpolation, use both of the addition pixel value and the pixelvalue of the normal pixel in the interpolation, or use only the pixelvalue of the normal pixel in the interpolation. The pixel value to beused in the interpolation can be appropriately decided depending on anaspect, a scene, and the like of the interpolation.

It is preferable that the imaging apparatus according to still anotheraspect of the present invention further comprises a signal gradientcalculation unit that calculates signal gradients of the pixelssurrounding the pixel position of the in-focus pixel, and the firstinterpolation unit performs an interpolation operation on the pixelvalue at the pixel position of the in-focus pixel using a pixel value ofa pixel selected based on the calculated signal gradients out of thesurrounding pixels.

According to still another aspect of the present invention, a pixel usedfor interpolating the in-focus pixel is selected based on the signalgradients of the pixels surrounding the pixel position of the in-focuspixel which is the correction target. Thus, the correction accuracy ofthe phase difference pixel can be increased. For example, erroneousinterpolation can be prevented (interpolation accuracy can be improved)by detecting a plurality of pixels that have the same color as a colorat the pixel position of the in-focus pixel and are present in a signalgradient direction in which the signal gradients of the surroundingpixels are minimized, and using pixel values of the plurality ofdetected pixels in the interpolation. Four directions including thefirst direction, the second direction, and a third direction and afourth direction between the first direction and the second directionare considered as the signal gradient direction.

It is preferable that the imaging apparatus according to still anotheraspect of the present invention further comprises a saturationdetermination unit that determines saturation of at least one pixel ofthe in-focus pixel, the first phase difference pixel or the second phasedifference pixel adjacent to the in-focus pixel, or the normal pixelwhich is adjacent to the in-focus pixel and in which the first filter isarranged, and in a case where the saturation determination unitdetermines that the pixel is saturated, the first interpolation unituses only the pixel value of the normal pixel in which the first filteris arranged.

In a case where the normal pixel which is adjacent to the in-focus pixeland in which the first filter is arranged is saturated, the additionpixel is generally also saturated. In a case where the addition pixelvalue of the addition pixel is clipped based on a saturation level, thepixel values of the saturated normal pixels match, and a problem doesnot particularly arise. In a case where the value of the leveladjustment coefficient for adjusting the addition pixel value cannot becorrectly set, the addition pixel adjusted using the level adjustmentcoefficient may not be saturated. In this case, a level difference insignal is present between the adjusted addition pixel and the normalpixel. In such a case, the addition pixel should not be used in theinterpolation.

In addition, even in a case where the normal pixel which is adjacent tothe in-focus pixel and in which the first filter is arranged is notsaturated, the in-focus pixel and the first phase difference pixel orthe second phase difference pixel adjacent to the in-focus pixel may besaturated in a case where high frequency strong light is incident onthese pixels. In this case, the addition pixel value is not reliable,and the addition pixel should not be used in the interpolation. Thesaturation of the pixel refers to a case of exceeding a presetsaturation level and is not necessarily limited to a maximum value thatcan be output from the pixel of the imaging element.

It is preferable that the imaging apparatus according to still anotheraspect of the present invention further comprises a second interpolationunit that sets the first phase difference pixel or the second phasedifference pixel as the in-focus pixel and generates the pixel value atthe pixel position of the in-focus pixel by gain interpolation, thesecond interpolation unit generating the pixel value at the pixelposition of the in-focus pixel by the gain interpolation based on thepixel value of the in-focus pixel and gain interpolation information setfor the pixel position of the in-focus pixel.

Approximately half of the intensity of light incident on the surroundingnormal pixel is incident on the first phase difference pixel and thesecond phase difference pixel. Thus, sensitivity is decreased below thatof the normal pixel. The “gain interpolation” refers to interpolation inwhich a signal level is adjusted to that of the normal pixel bymultiplying the pixel value of the phase difference detection pixel bypredetermined gain interpolation information in order to supplement thedecrease in sensitivity of the phase difference detection pixel. In thecase of interpolating the phase difference detection pixel, depending onan imaging scene and the like, it may be more appropriate to perform thegain interpolation than “average value interpolation” that uses thesurrounding pixel of the in-focus pixel. In that case, the “gaininterpolation” is performed.

It is preferable that the imaging apparatus according to still anotheraspect of the present invention further comprises a final pixel valuedecision unit that decides a final pixel value at the pixel position ofthe in-focus pixel by selecting any one pixel value of two pixel valuesincluding the pixel value generated by the first interpolation unit atthe pixel position of the in-focus pixel and the pixel value generatedby the second interpolation unit at the pixel position of the in-focuspixel or generating a pixel value obtained by weighted addition of thetwo pixel values. Any one pixel value selected from the pixel valuesubjected to the “average interpolation” by the first interpolation unitand the pixel value subjected to the “gain interpolation” by the secondinterpolation unit or the pixel value obtained by the weighted additionof the two pixel values is set as the final pixel value at the pixelposition of the in-focus pixel.

In the imaging apparatus according to still another aspect of thepresent invention, it is preferable that in a case where the pixelposition of the in-focus pixel corresponds to a pixel position of thenormal pixel in which the second filter is arranged, the firstinterpolation unit uses only the normal pixel in which the second filteris arranged. The reason is that since the addition pixel value obtainedby adding the pixel values of the pair of the first phase differencepixel and the second phase difference pixel is the pixel value of thevirtual pixel including the first filter, the addition pixel valuecannot be used in the interpolation of the in-focus pixel correspondingto the second filter.

In the imaging apparatus according to still another aspect of thepresent invention, it is preferable that the first filter is a greenfilter allowing transmission in a wavelength range of green, and theplurality of second filters include a red filter allowing transmissionin a wavelength range of red and a blue filter allowing transmission ina wavelength range of blue, the first periodic color arrangementarranged in the two-dimensionally arranged plurality of phase differencedetection pixels and the plurality of normal pixels is configured byalternately arranging, in the first direction and the second direction,a first arrangement that corresponds to 3×3 pixels and in which thegreen filters are arranged at a center and four corners, the red filtersare arranged in the first direction on both sides of the green filter atthe center, and the blue filters are arranged in the second direction,and a second arrangement that corresponds to 3×3 pixels and in which thegreen filters are arranged at a center and four corners, the bluefilters are arranged in the first direction on both sides of the greenfilter at the center, and the red filters are arranged in the seconddirection, and the imaging element includes a phase difference pixel rowin which the first phase difference pixel and the second phasedifference pixel are arranged at positions adjacent to the green filterin the first direction, and a normal pixel row in which only the normalpixel is arranged in the first direction.

The imaging element having the first periodic color arrangement of theabove configuration includes 2×2 pixels in which pixels (G pixels)including the green filter are adjacently arranged. The first phasedifference pixel and the second phase difference pixel can be arrangedat the positions of two G pixels adjacent in the first direction in the2×2 pixels. Even in a case where the two G pixels of the 2×2 pixels areassigned to the first phase difference pixel and the second phasedifference pixel, G pixels (normal pixels) are present around the firstphase difference pixel and the second phase difference pixel. Thus, theaverage value interpolation can be performed with high interpolationaccuracy.

In the imaging apparatus according to still another aspect of thepresent invention, it is preferable that the first periodic colorarrangement arranged in the plurality of normal pixels is a Bayerarrangement, the imaging element includes a phase difference pixel rowin which the first phase difference pixel, the second phase differencepixel, and the normal pixel are arranged in the first direction, and anormal pixel row in which only the normal pixel is arranged in the firstdirection, and in the phase difference pixel row, three pixels includingthe first and second phase difference pixels and one normal pixel as onecycle are periodically arranged, and a green filter allowingtransmission in a wavelength range of green is arranged in the firstphase difference pixel and the second phase difference pixel.

In the imaging element having the Bayer arrangement, by disposing thephase difference pixel row in which three pixels including the first andsecond phase difference pixels and one normal pixel as one cycle areperiodically arranged, the normal pixel including the green filter (Gpixel) and the normal pixel including the blue filter (B pixel) areincluded in the phase difference pixel row in a case where the phasedifference pixel row is formed in a row (GB row) of the Bayerarrangement in which the green filter and the blue filter arealternately arranged. Thus, the average value interpolation can beaccurately performed.

In addition, in the imaging element having the Bayer arrangement, the Gpixels corresponding to the first color most contributing to obtaining abrightness signal are arranged more than (as twice as) the B pixels orthe normal pixels including the red filter (R pixels). Thus, the imagingelement having the Bayer arrangement is the most general imaging elementcapable of improving resolution, improving reproduction of a highbrightness component, and reducing jaggedness. By arranging the greenfilter in the first phase difference pixel and the second phasedifference pixel, the average value interpolation of the in-focus pixelcan be accurately performed using the pixel values of the surrounding Gpixels (the number of G pixel is large) and/or the addition pixel valuein a case where the in-focus pixel is at a position corresponding to theG pixel.

It is preferable that the imaging apparatus according to still anotheraspect of the present invention further comprises an imaging opticalsystem that forms a subject image on a light-receiving surface of theimaging element, a phase difference detection unit that detects a phasedifference between a first pixel value obtained from the first phasedifference pixel of the imaging element and a second pixel valueobtained from the second phase difference pixel, and an autofocuscontrol unit that controls the imaging optical system based on the phasedifference detected by the phase difference detection unit.

Since the opening portions of the pair of the first phase differencepixel and the second phase difference pixel are adjacently arranged toface each other, an interval between the pair of the first phasedifference pixel and the second phase difference pixel is minimized.Accordingly, a spatial sampling frequency of a phase difference can bemaximized, and phase difference AF for a subject having a high spatialfrequency can be performed more favorably (with higher accuracy) thanthat in a case where the pair of the first phase difference pixel andthe second phase difference pixel is separately arranged with the normalpixel interposed therebetween.

Still another aspect of the invention is an image processing method foran imaging apparatus comprising an imaging element in which a pluralityof phase difference detection pixels and a plurality of normal pixelsare two-dimensionally arranged in a first direction and a seconddirection orthogonal to the first direction, the phase differencedetection pixel includes a first phase difference pixel and a secondphase difference pixel including opening portions for pupil separationat different positions in the first direction, and the opening portionsof the first phase difference pixel and the second phase differencepixel are adjacently arranged to face each other, in the plurality ofnormal pixels, a first filter corresponding to a first color mostcontributing to obtaining a brightness signal and a plurality of secondfilters respectively corresponding to two or more colors other than thefirst color are arranged in a first periodic color arrangement, and ineach of the first phase difference pixel and the second phase differencepixel, the first filter is arranged, or light in a wavelength rangewider than a transmission wavelength range of the first filter isincident. The method comprises a step of generating an addition pixelvalue at a pixel position between the first phase difference pixel andthe second phase difference pixel by adding pixel values of the firstphase difference pixel and the second phase difference pixel of whichthe opening portions are adjacently arranged to face each other, a stepof selecting the first phase difference pixel or the second phasedifference pixel that is not processed as an in-focus pixel, and aninterpolation step of generating a pixel value at a pixel position ofthe selected in-focus pixel by using pixel values of pixels surroundingthe pixel position of the in-focus pixel, the addition pixel value beingused as a pixel value of one pixel of the surrounding pixels.

It is preferable that the image processing method according to stillanother aspect of the present invention further comprises a step ofcorrecting the addition pixel value by multiplying the addition pixelvalue by a set level adjustment coefficient, and in the interpolationstep, the addition pixel value corrected using the level adjustmentcoefficient is used.

In the image processing method according to still another aspect of thepresent invention, it is preferable that in the interpolation step, atleast one pixel value of the addition pixel value or a pixel value ofthe normal pixel in which the first filter is arranged is used.

It is preferable that the image processing method according to stillanother aspect of the present invention further comprises a step ofcorrecting the addition pixel value by multiplying the addition pixelvalue by a set level adjustment coefficient, and in the interpolationstep, an interpolation operation is performed on the pixel value at thepixel position of the in-focus pixel using a pixel value of a pixelselected based on the calculated signal gradients out of the surroundingpixels.

According to the present invention, in the case of interpolating thepixel value for imaging at the pixel position of the first phasedifference pixel or the second phase difference pixel (in-focus pixel),which is the correction target, using the pixel values of the pixelssurrounding the in-focus pixel, the virtual pixel is created between thepair of the first phase difference pixel and the second phase differencepixel by adding the pixel values of the pair of the first phasedifference pixel and the second phase difference pixel of which theopening portions are adjacently arranged to face each other. The pixelvalue (addition pixel value) of the virtually created pixel is used inthe interpolation. Thus, the correction accuracy of the in-focus pixelcan be increased. Accordingly, even in a case where the phase differencedetection pixel is densely arranged in the imaging element in order tosecure AF performance, the interpolation accuracy decreased by the densearrangement can be complemented using the addition pixel value.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a perspective view illustrating one example of an imagingapparatus.

FIG. 2 is a rear view of the imaging apparatus illustrated in FIG. 1.

FIG. 3 is a block diagram illustrating one example of an internalconfiguration of the imaging apparatus illustrated in FIG. 1.

FIG. 4 is a diagram illustrating a first embodiment of color filterarrangement and arrangement of phase difference detection pixels in animaging element.

FIG. 5 is a diagram in which a basic arrangement pattern P illustratedin FIG. 4 is divided into four 3×3 pixels.

FIG. 6 is a plan view schematically illustrating a pair of a first phasedifference pixel PR and a second phase difference pixel PL.

FIG. 7 is an enlarged main view illustrating configurations of the firstphase difference pixel PR and the second phase difference pixel PL.

FIG. 8 is a graph illustrating sensitivities of a normal pixel (Gpixel), the first phase difference pixel PR, and the second phasedifference pixel PL in a left-right direction of the imaging elementunder a certain condition.

FIG. 9 is a block diagram illustrating a first embodiment of aninterpolation processing unit in an image processing unit 24 illustratedin FIG. 3.

FIG. 10 is a diagram for describing average value interpolation for thephase difference detection pixels in the imaging element of the firstembodiment.

FIG. 11 is a block diagram illustrating a second embodiment of theinterpolation processing unit in the image processing unit 24illustrated in FIG. 3.

FIG. 12 is a block diagram illustrating a third embodiment of theinterpolation processing unit in the image processing unit 24illustrated in FIG. 3.

FIG. 13 is a block diagram illustrating a fourth embodiment of theinterpolation processing unit in the image processing unit 24illustrated in FIG. 3.

FIG. 14 is a diagram illustrating a 5×5 pixel window with an in-focuspixel (first phase difference pixel PR) at the center and a plurality ofG pixels (G1 to G10) in the window.

FIG. 15 is a flowchart illustrating an image processing method accordingto a first embodiment of the present invention.

FIG. 16 is a flowchart illustrating an image processing method accordingto a second embodiment of the present invention.

FIG. 17 is a diagram illustrating a second embodiment of the colorfilter arrangement and the arrangement of phase difference detectionpixels in the imaging element.

FIG. 18 is a diagram for describing the average value interpolation forthe phase difference detection pixels in the imaging element of thesecond embodiment.

FIG. 19 is another diagram for describing the average valueinterpolation for the phase difference detection pixels in the imagingelement of the second embodiment.

FIG. 20 is a diagram illustrating an exterior of a smartphone as oneembodiment of the imaging apparatus.

FIG. 21 is a block diagram illustrating an internal configuration of asmartphone 100 illustrated in FIG. 20.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereinafter, an imaging apparatus and an image processing methodaccording to preferred embodiments of the present invention will bedescribed in accordance with the appended drawings.

[Imaging Apparatus]

FIG. 1 and FIG. 2 are respectively a perspective view and a rear viewillustrating the exterior of the imaging apparatus according to theembodiment of the present invention. An imaging apparatus 10 is adigital camera that receives light passing through a lens by an imagingelement, converts the light into a digital signal, and records thedigital signal in a recording medium as image data of a still picture ora motion picture.

As illustrated in FIG. 1, in the imaging apparatus 10, an imaging lens12, a strobe 1, and the like are arranged on a front surface, and ashutter button 2, a power supply/mode switch 3, a mode dial 4, and thelike are arranged on an upper surface. As illustrated in FIG. 2, aliquid crystal monitor 30, a zoom button 5, a cross button 6, a MENU/OKbutton 7, a playback button 8, a BACK button 9, and the like arearranged on the rear surface of the camera.

The imaging lens 12 is composed of a retractable zoom lens and iswithdrawn from the main body of the camera by setting an operation modeof the camera to an imaging mode by the power supply/mode switch 3. Thestrobe 1 radiates strobe light to a main subject.

The shutter button 2 is configured with a so-called stroke type switchof two stages including “half push” and “full push” and functions as animaging preparation instruction unit and also functions as an imagerecording instruction unit.

In a case where a still picture imaging mode is selected as the imagingmode and the shutter button 2 is subjected to the “half push”, theimaging apparatus 10 performs an imaging preparation operation ofperforming autofocus (AF)/auto exposure (AE) control. In a case wherethe shutter button 2 is subjected to the “full push”, the imagingapparatus 10 images and records a still picture.

In addition, in a case where a motion picture imaging mode is selectedas the imaging mode and the shutter button 2 is subjected to the “fullpush”, the imaging apparatus 10 starts recording a motion picture. In acase where the shutter button 2 is subjected to the “full push” again,the imaging apparatus 10 stops recording and enters a standby state.

The power supply/mode switch 3 has a function as a power supply switchfor setting a power supply of the imaging apparatus 10 to be ON/OFF anda function as a mode switch for setting the mode of the imagingapparatus 10. The power supply/mode switch 3 is arranged to be slidableamong an “OFF position”, a “playback position”, and an “imagingposition”. The power supply of the imaging apparatus 10 is switched ONby sliding the power supply/mode switch 3 to the “playback position” orthe “imaging position”. The power supply of the imaging apparatus 10 isswitched OFF by sliding the power supply/mode switch 3 to the “OFFposition”. Sliding the power supply/mode switch 3 to the “playbackposition” sets a “playback mode”, and sliding the power supply/modeswitch 3 to the “imaging position” sets the “imaging mode”.

The mode dial 4 functions as a mode switching unit that sets the imagingmode of the imaging apparatus 10. The imaging mode of the imagingapparatus 10 is set to various modes depending on a setting position ofthe mode dial 4. For example, the “still picture imaging mode” forperforming still picture imaging and the “video imaging mode” forperforming motion picture imaging are present.

The liquid crystal monitor 30 displays a live view image at the time ofthe imaging mode, displays a still picture or a motion picture at thetime of the playback mode, and functions as a part of a graphical userinterface by displaying a menu screen and the like.

The zoom button 5 functions as zoom instruction means for providing aninstruction to zoom and includes a tele button 5T providing aninstruction to zoom to a telephoto side and a wide button 5W providingan instruction to zoom to a wide angle side. In the imaging apparatus10, operating the tele button 5T and the wide button 5W at the time ofthe imaging mode changes the focal length of the imaging lens 12. Inaddition, operating the tele button 5T and the wide button 5W at thetime of the playback mode enlarges and shrinks the image in playback.

The cross button 6 is an operation unit that inputs instructions in fourdirections of upward, downward, leftward, and rightward directions andfunctions as a button (cursor movement operation means) for selecting anitem from the menu screen or providing an instruction to select varioussetting items from each menu. A left/right key functions as a button(forward direction/backward direction forwarding) for frame forwardingat the time of the playback mode.

The MENU/OK button 7 is an operation button having functions of both amenu button for providing an instruction to display the menu on thescreen of the liquid crystal monitor 30 and an OK button for providingan instruction to confirm and execute the content of selection and thelike.

The playback button 8 is a button for switching to the playback mode inwhich the imaged and recorded still picture or motion picture isdisplayed on the liquid crystal monitor 30.

The BACK button 9 functions as a button for providing an instruction tocancel an input operation and return to the immediately previousoperation state.

In the imaging apparatus 10 according to the present embodiment, thefunctions of the buttons/switches may be implemented by disposing andoperating a touch panel without disposing members specific to thebuttons/switches.

[Internal Configuration of Imaging Apparatus]

FIG. 3 is a block diagram illustrating an embodiment of an internalconfiguration of the imaging apparatus 10. The imaging apparatus 10records the captured image in a memory card 54. The operation of thewhole apparatus is managed and controlled by a central processing unit(CPU) 40.

An operation unit 38 such as the shutter button 2, the power supply/modeswitch 3, the mode dial 4, the tele button 5T, the wide button 5W, thecross button 6, the MENU/OK button 7, the playback button 8, and theBACK button 9 is disposed in the imaging apparatus 10. A signal from theoperation unit 38 is input into the CPU 40. The CPU 40 controls eachcircuit of the imaging apparatus 10 based on the input signal. Forexample, the CPU 40 performs drive control of the imaging element, lensdrive control, stop drive control, imaging operation control, imageprocessing control, recording/playback control of the image data, anddisplay control of the liquid crystal monitor 30.

In a case where the power supply of the imaging apparatus 10 is switchedON by the power supply/mode switch 3, power is supplied to each blockfrom a power supply unit, not illustrated, and the imaging apparatus 10starts to be driven.

An image of a luminous flux passing through the imaging lens 12, a stop14, a mechanical shutter 15, and the like is formed in an imagingelement 16 that is a complementary metal-oxide semiconductor (CMOS) typecolor image sensor. The imaging element 16 is not limited to a CMOS typeand may be a color image sensor of an XY address type or a chargecoupled device (CCD) type.

In the imaging element 16, multiple light-receiving elements(photodiodes) are two-dimensionally arranged. A subject image formed ona light-receiving surface of each photodiode is converted into a signalvoltage (or charge) of an amount corresponding to the intensity of anincidence ray. The signal voltage is converted into a digital signalthrough an analog/digital (A/D) converter in the imaging element 16 andis output.

<First Embodiment of Imaging Element>

In the imaging element 16, color filters of red (R), green (G), and blue(B) are arranged in a first periodic color arrangement, illustratedbelow, on a plurality of pixels configured with photoelectric conversionelements (photodiodes) that are two-dimensionally arranged in a firstdirection (horizontal direction) and a second direction (verticaldirection) orthogonal to the first direction.

In addition, in the imaging element 16, a plurality of phase differencedetection pixels and a plurality of normal pixels (pixels other than thephase difference detection pixel) for imaging are arranged.

FIG. 4 is a diagram illustrating a first embodiment of color filterarrangement and arrangement of the phase difference detection pixels inthe imaging element 16.

As illustrated in FIG. 4, a first filter corresponding to a first color(green) and any of a plurality of second filters respectivelycorresponding to two or more colors (red and blue) other than green arearranged in a first periodic color arrangement in the plurality ofnormal pixels of the imaging element 16.

The first periodic color arrangement of the color filters of the imagingelement 16 of the first embodiment is the X-Trans (registered trademark)arrangement.

In the X-Trans arrangement, the red filter (R filter) allowingtransmission in a wavelength range of red (R), the blue filter (Bfilter) allowing transmission in a wavelength range of blue (B), and thegreen filter (G filter) allowing transmission in a wavelength range ofgreen (G) are arranged with predetermined periodicity. The G filtercorresponds to a first filter that corresponds to a first color mostcontributing to obtaining a brightness signal compared to a second color(in this embodiment, colors of R and B). The R filter and the B filtercorrespond to a plurality of second filters that respectively correspondto two or more colors other than the first color.

The X-Trans arrangement includes a basic arrangement pattern P (patternillustrated by a thick frame) including a square arrangement patterncorresponding to 6×6 pixels. The basic arrangement pattern P isrepeatedly arranged in the horizontal direction and the verticaldirection.

FIG. 5 illustrates a state where the basic arrangement pattern Pillustrated in FIG. 4 is divided into four 3×3 pixels.

As illustrated in FIG. 5, the basic arrangement pattern P is anarrangement in which an A arrangement (first arrangement) of 3×3 pixelssurrounded by a solid line frame and a B arrangement (secondarrangement) of 3×3 pixels surrounded by a broken line frame arealternately arranged in the horizontal direction and the verticaldirection.

In the A arrangement, the G filters are arranged at the center and thefour corners of 3×3 pixels. The R filters are arranged in the horizontaldirection on both sides of the G filter. The B filters are arranged inthe vertical direction.

In the B arrangement, the G filters are arranged at the center and thefour corners of 3×3 pixels. The B filters are arranged in the horizontaldirection on both sides of the G filter. The R filters are arranged inthe vertical direction.

The basic arrangement pattern P includes the G filters of a squarearrangement corresponding to 2×2 pixels. The G filters of the squarearrangement corresponding to 2×2 pixels is are formed by arranging the Gfilters at the four corners and the center of 3×3 pixels in the Aarrangement or the B arrangement and alternately arranging the 3×3pixels in the horizontal direction and the vertical direction.

The imaging element 16 includes a phase difference pixel row in whichthe phase difference detection pixels are arranged, and a normal pixelrow in which only the normal pixels are arranged. In the exampleillustrated in FIG. 4, the eighth row corresponds to the phasedifference pixel row. While only one phase difference pixel row isillustrated in FIG. 4, the phase difference pixel row is disposed at acertain interval (on both sides of a plurality of normal pixel rows) onthe whole surface of the sensor or a specific AF region.

In addition, the phase difference pixel row is disposed in a row inwhich two pixels including the G filters (G pixels) are continuous. Thephase difference detection pixel is arranged at the position of thecontinuous G pixels.

FIG. 6 is a plan view schematically illustrating the phase differencedetection pixel (phase difference detection pixel illustrated by a thickframe A in FIG. 4) in the phase difference pixel row (eighth row)illustrated in FIG. 4.

As illustrated in FIG. 6, the phase difference detection pixel has anopening portion for pupil separation and includes a first phasedifference pixel PR and a second phase difference pixel PL havingopening portions at different positions in the horizontal direction. Theopening portions of a pair of the first phase difference pixel PR andthe second phase difference pixel PL are adjacently arranged to faceeach other.

The first phase difference pixel PR is a right opening pixel having theopening portion in the right half of the pixel. The second phasedifference pixel PL is a left opening pixel having the opening portionin the left half of the pixel.

FIG. 7 is an enlarged main view illustrating configurations of the firstphase difference pixel PR and the second phase difference pixel PL.

As illustrated in FIG. 7, a light shielding film 16A is arranged on thefront surface side (microlens L side) of a photodiode PD of the firstphase difference pixel PR, and a light shielding film 16B is arranged onthe front surface side of the photodiode PD of the second phasedifference pixel PL. The microlens L and the light shielding films 16Aand 16B have a pupil separation function. In FIG. 6, the light shieldingfilm 16A shields the left half of the light-receiving surface of thephotodiode PD from light. Thus, the first phase difference pixel PRreceives only a luminous flux passing on the left side of an opticalaxis among luminous fluxes passing through an exit pupil of the imaginglens 12. In addition, the G filter is arranged below the microlens L asa color filter CF.

The light shielding film 16B shields the right half of thelight-receiving surface of the photodiode PD of the second phasedifference pixel PL from light. Thus, the second phase difference pixelPL receives only a luminous flux passing on the right side of theoptical axis among the luminous fluxes passing through the exit pupil ofthe imaging lens 12. By the microlens L and the light shielding films16A and 16B having the pupil separation function, the luminous fluxespassing through the exit pupil on the left and right sides are separatedand are respectively incident on the first phase difference pixel PR andthe second phase difference pixel PL.

FIG. 8 is a graph illustrating sensitivities of the normal pixel (Gpixel), the first phase difference pixel PR, and the second phasedifference pixel PL in a left-right direction of the imaging element 16under a certain condition.

As illustrated in FIG. 8, the sensitivity of the normal pixel (G pixel)of which the opening portion is not shielded from light has the highestsensitivity. The sensitivities of the first phase difference pixel PRand the second phase difference pixel PL are lower than the sensitivityof the G pixel. In addition, the left half of the opening of the firstphase difference pixel PR and the right half of the opening of thesecond phase difference pixel PL are shielded from light by the lightshielding films. Thus, the sensitivities (signal values) of the firstphase difference pixel PR and the second phase difference pixel PL areleft-right asymmetric with the center of the sensor (=image) as areference.

The sensitivities (signal values) of the first phase difference pixel PRand the second phase difference pixel PL match at the center of thesensor. The sensitivity of the first phase difference pixel PR isincreased (signal value is increased) in the direction of the left endside of the sensor. The sensitivity of the second phase difference pixelPL is increased (signal value is increased) in the direction of theright end side of the sensor.

In a case where a sensor position x is focused in FIG. 8, a relationshipamong the signal value (G) of the G pixel and the signal values (PR andPL) of the first phase difference pixel PR and the second phasedifference pixel PL at the sensor position x satisfies G>PL>PR.

An addition pixel value (PA (=PL+PR)) obtained by adding the signalvalues (pixel values) of the pair of the first phase difference pixel PRand the second phase difference pixel PL almost matches the signal value(G) of the normal pixel (G pixel) disposed at the same position (PA G)regardless of the sensor position at which the pair of the first phasedifference pixel PR and the second phase difference pixel PL isdisposed.

That is, as illustrated in FIG. 6, in a case where the image signals(pixel values) of the pair of the first phase difference pixel PR andthe second phase difference pixel PL are added, the added pixel value(addition pixel value) is almost equal to the pixel value of the normalpixel (G pixel). In addition, the added pixel (addition pixel) can beregarded as being present between the pair of the first phase differencepixel PR and the second phase difference pixel PL.

The pixel value (in the present example, corresponds to the pixel valueof the G pixel of the normal pixel) of the addition pixel including thepair of the first phase difference pixel PR and the second phasedifference pixel PL can be used in the case of performing aninterpolation operation on the pixel value of the G pixel at the pixelposition of the in-focus pixel of the first phase difference pixel PR orthe second phase difference pixel PL using the average valueinterpolation. Details of the correction of the phase difference valuewill be described below.

Returning to FIG. 3, the image signal (image data) read from the imagingelement 16 at the time of imaging the motion picture or the stillpicture is temporarily stored in a memory (synchronous dynamic randomaccess memory (SDRAM)) 48 or is input into a phase difference detectionunit 42, an AE detection unit 44, and the like through an image inputcontroller 22.

The CPU 40 manages and controls each unit of the imaging apparatus 10based on the operation in the operation unit 38 and performs an AFoperation and an AE operation at all times during imaging (display) ofthe live view image and imaging (recording) of the motion picture.

The phase difference detection unit 42 is a part performing the phasedifference AF process and detects the phase difference using the outputsignal of each of the first phase difference pixel PR and the secondphase difference pixel PL obtained through the image input controller22. Details of the detection of the phase difference by the phasedifference detection unit 42 will be described below.

In a case where phase difference data indicating the phase difference isinput from the phase difference detection unit 42, the CPU 40 functionsas a focal point adjusting unit that performs the phase difference AFbased on the phase difference data. That is, the CPU 40 calculates adeviation amount (defocus amount) between a focus position of theimaging lens 12 and an image forming surface of the imaging element 16based on the phase difference data and moves a focus lens in the imaginglens 12 through a lens drive unit 36 such that the calculated defocusamount becomes zero. The calculation of the defocus amount may beperformed by the phase difference detection unit 42.

The AE detection unit 44 calculates the integrating accumulation of theimage data (for example, the pixel values of the G pixels of the wholescreen) obtained through the image input controller 22 or calculates theintegrating accumulation of the image data (pixel values of the Gpixels) differently weighted between a center portion and a peripheralportion of the screen and outputs the integrating accumulation value tothe CPU 40. The CPU 40 calculates the brightness (imaging exposure value(Ev value)) of the subject from the integrating accumulation value inputfrom the AE detection unit 44. In a case where the imaging mode is thestill picture imaging mode, the CPU 40 performs the above AF controlagain in a case where the shutter button 2 is subjected to a first stagepush (half push). In a case where the shutter button 2 is subjected tothe full push, the CPU 40 calculates the brightness (imaging Ev value)of the subject, decides the F-number of the stop 14 and a light exposuretime (shutter speed) of the mechanical shutter 15 based on thecalculated imaging Ev value, and images the still picture (exposurecontrol).

In a case where the imaging mode is the motion picture imaging mode, theCPU 40 starts imaging and recording (picture recording) the motionpicture in a case where the shutter button 2 is subjected to the fullpush. At the time of imaging the motion picture, the CPU 40 opens themechanical shutter 15, consecutively reads (for example, 30frames/second or 60 frames/second as a frame rate) the image data fromthe imaging element 16, consecutively performs the phase difference AF,calculates the brightness of the subject, and controls the shutter speed(a charge accumulation time by rolling shutter) by a shutter drive unit33 and/or the stop 14 by a stop drive unit 34.

The CPU 40 operates the zoom lens to advance and retract in the opticalaxis direction through the lens drive unit 36 in response to the zoominstruction from the zoom button 5 and changes the focal length.

In addition, the ROM 47 is a read only memory (ROM) or an electricallyerasable programmable read-only memory (EEPROM) storing defectinformation related to the imaging element 16 and various parameters andtables used in image processing and the like. In the present example,the ROM 47 stores information related to the phase difference pixel row(including the pixel positions of the first phase difference pixel PRand the second phase difference pixel PL) and the normal pixel row ofthe imaging elements 16 and gain interpolation information, a leveladjusting coefficient, and the like described below.

The image processing unit 24 reads non-processed image data (RAW data)temporarily stored in the memory 48 through the image input controller22 at the time of imaging the motion picture or the still picture. Theimage processing unit 24 performs an offset process, a pixelinterpolation process (interpolation process for the phase differencedetection pixel, a defective pixel, and the like), white balancecorrection, a gain control process including sensitivity correction,gamma-correction processing, demosaicing (referred to as a “demosaicingprocess”), a brightness and color difference signal generation process,a contour highlighting process, color correction, and the like on theread RAW data.

The image data processed as the live view image by the image processingunit 24 is input into a video RAM random access memory (VRAM) 50.

The VRAM 50 includes an A region and a B region. In each of the A regionand the B region, image data representing an image of one frame isrecorded. In the VRAM 50, the image data representing the image of oneframe is alternately rewritten between the A region and the B region.The written image data is read from a region of the A region and the Bregion of the VRAM 50 other than a region in which the image data isrewritten.

The image data read from the VRAM 50 is encoded in a video encoder 28and is output to the liquid crystal monitor 30 disposed on the rearsurface of the camera. Accordingly, the live view image showing thesubject image is displayed on the liquid crystal monitor 30.

The image data (brightness data (Y) and color difference data (Cb) and(Cr)) processed as the still picture or the motion picture for recordingby the image processing unit 24 is stored in the memory 48 again.

A compression/expansion processing unit 26 performs a compressionprocess on the brightness data (Y) and the color difference data (Cb)and (Cr) processed by the image processing unit 24 and stored in thememory 48 at the time of recording the still picture or the motionpicture. In the case of the still picture, for example, the compressionis performed in the Joint Photographic Experts Group (JPEG) format. Inthe case of the motion picture, for example, the compression isperformed in the H.264 format. The compression image data compressed bythe compression/expansion processing unit 26 is recorded in the memorycard 54 through a media controller 52.

In addition, the compression/expansion processing unit 26 performs anexpansion process on the compression image data obtained from the memorycard 54 through the media controller 52 at the time of the playbackmode. For example, the media controller 52 records and reads thecompression image data in the memory card 54.

[Phase Difference AF]

In the case of performing the phase difference AF, the CPU 40functioning as an autofocus control unit outputs a read instruction forreading the image data of the phase difference pixel row in at least anAF region of the imaging element 16 to a sensor drive unit 32 and readsthe corresponding image data from the imaging element 16.

At the time of imaging and displaying the motion picture (including thelive view image), the CPU 40 obtains a thinning-out rate for reading theimage data from the imaging element 16 in a thinned-out manner. Thethinning-out rate may be a preset fixed value or may be able to beselected by a user from a plurality of thinning-out rates. For example,the optimal thinning-out rate can be set in connection with selection ofthe image size of the motion picture or selection of the frame rate. Itis preferable that the rows read in a thinned-out manner include thephase difference pixel row.

The CPU 40 outputs a read instruction indicating a thinning-out pattern(extraction pattern) corresponding to the thinning-out rate to thesensor drive unit 32 and reads the image data from the imaging element16 in a thinned-out manner.

The phase difference detection unit 42 extracts output data of the phasedifference detection pixel (the first phase difference pixel PR and thesecond phase difference pixel PL) in the AF region from the read phasedifference pixel row and detects the phase difference between the outputdata (first pixel value) of the first phase difference pixel PR and theoutput data (second pixel value) of the second phase difference pixelPL. For example, the phase difference is obtained from a shift amount inthe left-right direction between the first pixel value and the secondpixel value when the correlation between the first pixel value and thesecond pixel value of the pair of the first phase difference pixel PRand the second phase difference pixel PL is the maximum (when theintegrating accumulation value of the absolute value of the differencebetween the pixel values of the pair of the phase difference pixels isthe minimum).

A value obtained by correcting the obtained shift amount by a positionaldeviation in the horizontal direction between the pair of the firstphase difference pixel PR and the second phase difference pixel PL canbe calculated as the phase difference data. A method of calculating thephase difference is not limited to the above method, and various methodscan be applied.

Next, the CPU 40 calculates the deviation amount (defocus amount)between the focus position of the imaging lens 12 (imaging opticalsystem) and the image forming surface of the imaging element 16 based onthe phase difference data detected by the phase difference detectionunit 42. The calculation of the defocus amount may be performed by thephase difference detection unit 42.

The CPU 40 performs the phase difference AF by moving the focus lens inthe imaging lens 12 through the lens drive unit 36 based on thecalculated defocus amount such that the defocus amount becomes zero.

In the imaging element 16, the opening portions of the pair of the firstphase difference pixel PR and the second phase difference pixel PL areadjacently arranged to face each other. Thus, the interval between thepair of the first phase difference pixel PR and the second phasedifference pixel PL is minimized. Accordingly, a spatial samplingfrequency of the phase difference can be maximized, and the phasedifference AF for the subject having a high spatial frequency can beperformed with higher accuracy than that in a case where the pair of thefirst phase difference pixel and the second phase difference pixel isseparately arranged on both sides of the normal pixel.

The rows read from the imaging element 16 in a thinned-out manner at thetime of generating the motion picture can include the phase differencepixel row including the phase difference detection pixel (the firstphase difference pixel PR and the second phase difference pixel PL). Thephase difference AF can be appropriately performed even during imagingof the motion picture.

[Interpolation Processing Unit]

<First Embodiment of Interpolation Processing Unit>

FIG. 9 is a block diagram illustrating a first embodiment of aninterpolation processing unit in the image processing unit 24illustrated in FIG. 3.

An interpolation processing unit 60 of the first embodiment illustratedin FIG. 9 is a part correcting (interpolating) the pixel value of thephase difference detection pixel (the first phase difference pixel PRand the second phase difference pixel PL) included in the image data(RAW data) read from the imaging element 16 at the time of switching tothe still picture imaging mode and imaging the still picture.

The interpolation processing unit 60 includes a gain interpolation unit61, an average value interpolation unit 62, a signal gradientcalculation unit 63, a pixel value addition unit 64, and a final pixelvalue decision unit 65.

Approximately half of the intensity of light incident on the surroundingnormal pixels is incident on the phase difference detection pixel (thefirst phase difference pixel PR and the second phase difference pixelPL). Thus, sensitivity is decreased below that of the normal pixel, andthe phase difference detection pixel cannot be used as the normal pixel.

The gain interpolation unit 61 functioning as a second interpolationunit performs interpolation in which a signal level is adjusted to thatof the normal pixel by multiplying the pixel value of the phasedifference detection pixel by predetermined gain interpolationinformation in order to supplement the decrease in sensitivity of thephase difference detection pixel.

The interpolation processing unit 60 includes a gain interpolationinformation obtaining unit that obtains the gain interpolationinformation set for the pixel position of the in-focus pixel in the RAWdata in a case where a correction target of the first phase differencepixel PR or the second phase difference pixel PL is set as the in-focuspixel.

The gain interpolation information obtaining unit may calculate the gaininterpolation information corresponding to the pixel position of thein-focus pixel based on the RAW data surrounding the in-focus pixel ormay obtain the gain interpolation information from a storage unit (ROM47) storing the gain interpolation information for each pixel positionof the in-focus pixel. The gain interpolation information can becalculated as the ratio of the pixel value of the in-focus pixel in theRAW data to the average pixel value of the normal pixels having the samecolor and surrounding the in-focus pixel.

The average value interpolation unit 62 functioning as a firstinterpolation unit is a part generating the pixel value at the pixelposition of the in-focus pixel using at least one of the pixel values ofthe normal pixels surrounding the pixel position of the in-focus pixelor the pixel value of the addition pixel of the pair of the first phasedifference pixel PR and the second phase difference pixel PL. Theaverage value interpolation unit 62 is provided with informationindicating a signal gradient direction calculated by the signal gradientcalculation unit 63 and the addition pixel value of the addition pixeladded by the pixel value addition unit 64.

The signal gradient calculation unit 63 calculates the signal gradientdirection in which the signal gradients of the surrounding pixels of thepixel position of the in-focus pixel are minimized in the case ofgenerating the pixel value at the pixel position of the in-focus pixel.

FIG. 10 is a diagram for describing the average value interpolation forthe phase difference detection pixel (first phase difference pixel PR)in the imaging element of the first embodiment.

In FIG. 10, in a case where the first phase difference pixel PR is setas the in-focus pixel, 10 G pixels (G1 to G10) are present in the rangeof 5×5 pixels with the in-focus pixel at the center. In addition, G11,G12, and G13 denote the addition pixels corresponding to the G pixels.

As illustrated in FIG. 10, in the case of calculating the signalgradient direction based on the pixels surrounding the in-focus pixel(first phase difference pixel PR), the signal gradient calculation unit63 obtains the pixel values of the G pixels surrounding the in-focuspixel and calculates the signal gradient of the horizontal directionfrom the pixel interval and the difference between the pixel values oftwo G pixels (for example, G4 and G5) in the horizontal direction (firstdirection). In the same manner, the signal gradient calculation unit 63calculates the signal gradient of the vertical direction from the pixelinterval and the difference between the pixel values of two G pixels(for example, G4 and G9) in the vertical direction (second direction),calculates the signal gradient of a +45 degree direction from the pixelinterval and the difference between the pixel values of two G pixels(for example, G2 and G5) in the +45 degree direction (third direction),and calculates the signal gradient of a −45 degree direction from thepixel interval and the difference between the pixel values of two Gpixels (for example, G1 and G4) in the −45 degree direction (fourthdirection). The pixels used in the calculation of the signal gradientsare not limited to the above example. For example, G pixels of 2×2pixels close to the in-focus pixel may be used.

The signal gradient calculation unit 63 calculates the direction of thesignal gradient in which the signal gradient is the minimum among theabove calculated signal gradients of the four directions as the signalgradient direction.

The use of the pixel value of the G pixel in the calculation of thesignal gradient direction is because the pixel value of the G pixel mostcontributes to obtaining the brightness signal (Y) among the pixelvalues of the R pixel, the G pixel, and the B pixel. The signal gradientdirection calculated in the above manner corresponds to a directionhaving the highest correlation with the brightness among the fourdirections.

The pixel value addition unit 64 generates the pixel value of thevirtual G pixel (addition pixel) at the pixel position between the firstphase difference pixel PR and the second phase difference pixel PL byadding the pixel values of the pair of the first phase difference pixelPR and the second phase difference pixel PL.

As described using FIG. 6 and FIG. 8, in a case where the pixel valuesof the pair of the first phase difference pixel PR and the second phasedifference pixel PL are added, the added pixel value (addition pixelvalue) is equal to the pixel value of the normal pixel (G pixel) at thesame pixel position. In addition, the addition pixel can be regarded asbeing present between the pair of the first phase difference pixel PRand the second phase difference pixel PL. The addition pixel value ofthe addition pixel generated by the pixel value addition unit 64 isoutput to the average value interpolation unit 62.

In the case of performing the interpolation operation on the pixel valueat the pixel position of the in-focus pixel, the average valueinterpolation unit 62 detects a plurality of G pixels (including theaddition pixel corresponding to the G pixel) that are present in thesignal gradient direction calculated by the signal gradient calculationunit 63 with the pixel position of the in-focus pixel as a reference andhave the same color as the color at the pixel position of the in-focuspixel, and generates the pixel value at the pixel position of thein-focus pixel by interpolating the pixel values of the plurality ofdetected G pixels.

As illustrated in FIG. 10, in a case where the signal gradient directionis the horizontal direction, the average value interpolation unit 62generates the pixel value at the pixel position of the in-focus pixel byinterpolating (calculating a weighted average depending on a distance)the pixel value of the addition pixel (G11 and G12) corresponding to theG pixel in the horizontal direction. In a case where the signal gradientdirection is the horizontal direction, the normal pixel (G pixel) is notpresent in the horizontal direction of the in-focus pixel. Thus, theaverage value interpolation uses only the addition pixel.

In a case where the signal gradient direction is the vertical direction,the average value interpolation unit 62 generates the pixel value at thepixel position of the in-focus pixel by interpolating two G pixels (G4and G9) in the vertical direction. In a case where the signal gradientdirection is the +45 degree direction, the average value interpolationunit 62 generates the pixel value at the pixel position of the in-focuspixel by interpolating two G pixels (G5 and G6) in the +45 degreedirection.

In addition, in a case where the signal gradient direction is the −45degree direction, the average value interpolation unit 62 can generatethe pixel value at the pixel position of the in-focus pixel byinterpolating two G pixels (G14 and G15) in the −45 degree direction. Inthis case, the G pixels of G14 and G15 are pixels (relatively separatedpixels) in the range of 5×5 pixels with the in-focus pixel at thecenter. Thus, it is considered that interpolation accuracy is decreased.

In a case where the signal gradient direction is the −45 degreedirection, the average value interpolation unit 62 may generate thepixel value at the pixel position of the in-focus pixel by interpolatingthe pixel values of two pixels of G4 and G11, three pixels of G4, G11,and G6, four pixels of G4, G11, G6, and G9, or the pixels of two sets ofG1 and G7 and G3 and G10 in the −45 degree direction. Alternatively, theaverage value interpolation unit 62 may employ the pixel value of theaddition pixel (G11).

In a case where the in-focus pixel is the second phase difference pixelPL, the average value interpolation unit 62 can perform theinterpolation operation on the pixel value for imaging at the pixelposition of the second phase difference pixel PL in the same manner asdescribed above. In addition, in the case of performing the averagevalue interpolation depending on the signal gradient direction, pixelsused in the average value interpolation are not limited to the aboveexample.

The final pixel value decision unit 65 decides the final pixel value atthe pixel position of the in-focus pixel by selecting any one of pixelvalues including the pixel value interpolated by the gain interpolationunit 61 and the pixel value interpolated by the average valueinterpolation unit 62 or generating a pixel value obtained by weightedaddition of two pixel values. For example, in a case where the imagesurrounding the in-focus pixel is even, the pixel value obtained by theaverage value interpolation is preferred. In a case where the spatialfrequency of the image surrounding the in-focus pixel is high, the pixelvalue obtained by the gain interpolation is preferred. In addition, in aregion out of focus, the pixel value obtained by the average valueinterpolation is preferred.

As described above, the interpolation processing unit 60 corrects(interpolates) the pixel value of the phase difference detection pixelincluded in the RAW data read from the imaging element 16 at the time ofimaging the still picture. Accordingly, the RAW data of the stillpicture in which the pixel value at the pixel position of the phasedifference detection pixel is corrected is generated.

<Second Embodiment of Interpolation Processing Unit>

FIG. 11 is a block diagram illustrating a second embodiment of theinterpolation processing unit in the image processing unit 24illustrated in FIG. 3. In FIG. 11, common parts in the first embodimentillustrated in FIG. 9 will be designated by the same reference signs,and detailed description of such parts will not be repeated.

The interpolation processing unit 60 of the second embodimentillustrated in FIG. 11 is different from the first embodimentillustrated in FIG. 9 in that an addition pixel level correction unit 66is added.

In FIG. 11, the addition pixel value (PA (=PL+PR)) obtained by addingthe pixel values of the pair of the first phase difference pixel PR andthe second phase difference pixel PL by the pixel value addition unit 64is a value close to the pixel value (G) of the normal pixel (G pixel) atthe same pixel position but does not match in a strict sense.

The addition pixel level correction unit 66 reads a level adjustmentcoefficient (K) corresponding to the pixel position of the in-focuspixel from a storage unit (ROM 47) storing the level adjustmentcoefficient in advance or calculates the level adjustment coefficient(K) by analyzing the image data. The addition pixel level correctionunit 66 multiplies the addition pixel value (PA) added by the pixelvalue addition unit 64 by the read or calculated level adjustmentcoefficient (K) and outputs the multiplied (level is adjusted) additionpixel value ((PA×K)=G) to the average value interpolation unit 62 as thepixel value (G) corresponding to the G pixel of the normal pixel.

According to the second embodiment, the addition pixel value is adjustedby multiplying the addition pixel value (PA (=PL+PR)) obtained by addingthe pixel values of the pair of the first phase difference pixel PR andthe second phase difference pixel PL by the level adjustment coefficient(K). Thus, the addition pixel value (PA×K) after adjustment can be setto completely match the pixel value (G) of the normal pixel (G pixel)that may be obtained in a case where the normal pixel is present at thesame pixel position. High accuracy interpolation can be performed in thecase of using the addition pixel value in the average valueinterpolation.

<Third Embodiment of Interpolation Processing Unit>

FIG. 12 is a block diagram illustrating a third embodiment of theinterpolation processing unit in the image processing unit 24illustrated in FIG. 3. In FIG. 12, common parts in the second embodimentillustrated in FIG. 11 will be designated by the same reference signs,and detailed description of such parts will not be repeated.

The interpolation processing unit 60 of the third embodiment illustratedin FIG. 12 is mainly different from the second embodiment illustrated inFIG. 11 in that a saturation determination unit 67 is added.

The saturation determination unit 67 determines saturation of at leastone of the in-focus pixel which is the interpolation target, the firstphase difference pixel PR or the second phase difference pixel PLadjacent to the in-focus pixel, or the normal pixel (G pixel) adjacentto the in-focus pixel. The saturation determination unit 67 outputs thedetermination result to the average value interpolation unit 62.

A case where strong light is incident on the G pixel adjacent to thein-focus pixel and the G pixel is saturated is considered.

In a case where the G pixel is saturated and any or both of the firstphase difference pixel PR and the second phase difference pixel PL arenot saturated, a relationship G<PA×K is generally satisfied. In a casewhere the addition pixel value (PA×K) after adjustment exceeds asaturation level, the addition pixel is set to match the G pixel byclipping the addition pixel value based on the saturation level, and aproblem does not particularly arise. However, in a case where the valueof the level adjustment coefficient (K) for adjusting the addition pixelvalue (PA) cannot be correctly set, the addition pixel value (PA×K)after adjustment may not exceed the saturation level even in a casewhere the G pixel is saturated. In this case, a level difference insignal is present between the addition pixel and the surrounding Gpixels. In such a case, the average value interpolation unit 62 shouldnot use the addition pixel in the average value interpolation.

As illustrated in FIG. 14, for example, when the first phase differencepixel PR is set as the in-focus pixel, the addition pixel adjacent tothe in-focus pixel is not used in the average value interpolation in acase where the surrounding G pixels, for example, any or all of the G4,G5, and G6, are saturated.

In addition, an assumption G=PA or G=PA×K corresponds to a case wherethe intensity of incidence ray on the G pixel and the intensity ofincidence ray on the first phase difference pixel PR and the secondphase difference pixel PL are constant (in a case where a monochromesubject is imaged). In a case where a high frequency signal pattern isimaged and high frequency strong light is incident on the in-focus pixeland the first phase difference pixel PR or the second phase differencepixel PL adjacent to the in-focus pixel, these pixels may be saturatedeven in a case where the G pixels adjacent to the in-focus pixel are notsaturated. In this case, the addition pixel value is not reliable, andthe assumption G=PA or G=PA×K is not established. Thus, the additionpixel should not be used in the interpolation.

Therefore, in a case where at least one of the first phase differencepixel PR or the second phase difference pixel PL exceeds a predeterminedsaturation level SATLEV, the addition pixel adjacent to the in-focuspixel is not used in the average value interpolation. For example, in acase where the depth of the pixels of the RAW data output from theimaging element 16 is 16 bits (representing 0 to 65535), SATLEV is setto 65000 or the like.

In a case where the determination result indicating that at least one ofthe in-focus pixel which is the interpolation target, the first phasedifference pixel PR or the second phase difference pixel PL adjacent tothe in-focus pixel, or the normal pixel (G pixel) adjacent to thein-focus pixel is saturated is input into the average valueinterpolation unit 62 from the saturation determination unit 67, theaverage value interpolation unit 62 does not use the addition pixel inthe average value interpolation. That is, in a case where the saturationdetermination unit 67 determines that at least one pixel of the in-focuspixel which is the interpolation target, the first phase differencepixel PR or the second phase difference pixel PL adjacent to thein-focus pixel, or the G pixel adjacent to the in-focus pixel issaturated, the average value interpolation unit 62 uses only the normalpixel (G pixel) in the average value interpolation.

According to the third embodiment, in the case of using the additionpixel in order to increase the accuracy of the average valueinterpolation, the addition pixel is not used in the average valueinterpolation on condition that using the addition pixel causes signalsaturation causing a decrease in image quality. Thus, the interpolationaccuracy of the average value interpolation can be maintained (imagequality can be maintained).

<Fourth Embodiment of Interpolation Processing Unit>

FIG. 13 is a block diagram illustrating a fourth embodiment of theinterpolation processing unit in the image processing unit 24illustrated in FIG. 3. In FIG. 13, common parts in the third embodimentillustrated in FIG. 12 will be designated by the same reference signs,and detailed description of such parts will not be repeated.

The interpolation processing unit 60 of the fourth embodimentillustrated in FIG. 13 is mainly different from the third embodimentillustrated in FIG. 12 in that a flatness determination unit 68 isadded.

The flatness determination unit 68 calculates flatness of an image in apredetermined window with the pixel position of the in-focus pixel as areference and determines whether or not the image in the window is flatusing the calculated flatness.

Information indicating the signal gradients of four directions (thehorizontal direction, the vertical direction, the +45 degree direction,and the −45 degree direction) calculated by the signal gradientcalculation unit 63 are input into the flatness determination unit 68 ofthe present example. The flatness determination unit 68 calculates themaximum signal gradient among the signal gradients of four directionsand determines that the image in the predetermined window is flat in acase where the calculated maximum signal gradient is less than or equalto a threshold (first threshold) for determining the flatness.

In addition, the flatness determination unit 68 is not limited to theabove example. As illustrated in FIG. 14, in a case where a range of 5×5pixels with the in-focus pixel (in FIG. 14, the first phase differencepixel PR) at the center is set as the predetermined window, the flatnessdetermination unit 68 may calculate a standard deviation or a varianceof the pixel values of a plurality of G pixels (10 G pixels of G1 toG10) in the window and may determine that the image in the window isflat in a case where the calculated standard deviation or variance isless than or equal to a threshold (second threshold) for determining theflatness. The size of the window is not limited to the range of 5×5pixels and can be set to M×N pixels (M and N are preferably odd numbersgreater than or equal to 3).

In a case where the determination result indicating that the image inthe predetermined window with the pixel position of the in-focus pixel,which is the interpolation target, as a reference is flat is input intothe average value interpolation unit 62 from the flatness determinationunit 68, the average value interpolation unit 62 does not use theaddition pixel in the average value interpolation. That is, in a casewhere the flatness determination unit 68 determines that the image inthe predetermined window is flat, the average value interpolation unit62 uses only the normal pixel (G pixel) in the average valueinterpolation.

In addition, in a case where the flatness determination unit 68determines that the image in the predetermined window is flat, theaverage value interpolation unit 62 may not use the pixel that is usedin the average value interpolation based on the signal gradientdirection calculated by the signal gradient calculation unit 63.Alternatively, the average value interpolation may be performed usingall G pixels or a part of G pixels in the window.

In a case where a level difference in signal is present between theaddition pixel and the normal pixel (in the present example, the Gpixel), using the addition pixel in the average value interpolationcauses the in-focus pixel subjected to the average value interpolationto have a different pixel value from the G pixel in the flat image.Particularly, the in-focus pixel is likely to stand out in the flatimage.

Therefore, in a case where the flatness determination unit 68 determinesthat the image in the predetermined window is flat, the average valueinterpolation unit 62 performs the average value interpolation withoutusing the addition pixel. In the case of performing the average valueinterpolation by weighting the normal pixel and the addition pixel, aweight for the addition pixel may be set to be small.

According to the fourth embodiment, in the case of using the additionpixel in order to increase the accuracy of the average valueinterpolation, the average value interpolation is performed using onlythe normal pixel (or by decreasing the weight of the addition pixel) ina flat portion of the image where erroneous correction is likely tostand out in a case where a level difference in signal is presentbetween the addition pixel and the normal pixel. Thus, a decrease inimage quality can be prevented.

[Image Processing Method]

FIG. 15 is a flowchart illustrating an image processing method accordingto a first embodiment of the present invention and particularly,illustrates a process procedure in the interpolation processing unit 60illustrated in FIG. 12.

In FIG. 15, the addition pixel level correction unit 66 illustrated inFIG. 12 sets the level adjustment coefficient (K) of the addition pixel(step S10). The addition pixel level correction unit 66 can set thepreset level adjustment coefficient (K) or the level adjustmentcoefficient (K) calculated by analyzing the RAW data.

In addition, the gain interpolation information obtaining unit includedin the gain interpolation unit 61 reads the gain interpolationinformation from the storage unit (ROM 47) storing the gaininterpolation information in advance or calculates the gaininterpolation information by analyzing the image data and sets the reador calculated gain interpolation information (step S12).

The interpolation processing unit 60 selects a non-interpolated phasedifference pixel (the first phase difference pixel PR or the secondphase difference pixel PL) as the in-focus pixel (step S14).

The signal gradient calculation unit 63 calculates the signal gradientdirections around the in-focus pixel selected in step S14 based on thepixel values of the G pixels surrounding the in-focus pixel (step S16).

In addition, the pixel value addition unit 64 adds the pixel values ofthe pair of the first phase difference pixel PR and the second phasedifference pixel PL. The addition pixel level correction unit 66 adjuststhe level of the addition pixel by multiplying the addition pixel valueof the addition pixel by the level adjustment coefficient set in stepS10 (step S18).

The gain interpolation unit 61 performs the gain interpolation in whichthe signal level is adjusted to that of the normal pixel by multiplyingthe pixel value of the in-focus pixel set in step S14 by the gaininterpolation information set in step S12 (step S20).

The average value interpolation unit 62 determines whether or not thepixel (at least one pixel of the in-focus pixel, the first phasedifference pixel PR or the second phase difference pixel PL adjacent tothe in-focus pixel, or the normal pixel (G pixel) adjacent to thein-focus pixel) is saturated based on the determination result inputfrom the saturation determination unit 67 (step S22). In a case where itis determined that the pixel is not saturated (in the case of “No”), thepixel value for imaging at the pixel position of the in-focus pixel iscalculated (subjected to the average value interpolation) using thenormal pixels (G pixels) surrounding the in-focus pixel and also theaddition pixel of which the level is adjusted in step S18 (step S24;interpolation step). In a case where it is determined that the pixel issaturated (in the case of “Yes”), the pixel value for imaging at thepixel position of the in-focus pixel is calculated (subjected to theaverage value interpolation) using only the normal pixels (G pixels)surrounding the in-focus pixel without using the addition pixel of whichthe level is adjusted in step S18 (step S26; interpolation step).

The final pixel value decision unit 65 selects any one pixel value ofthe pixel value subjected to the gain interpolation in step S20 or thepixel value subjected to the average value interpolation in step S24 orstep S26 or generates a pixel value obtained by weighted addition of thetwo pixel values, and decides the final interpolation value (final pixelvalue) at the pixel position of the in-focus pixel (step S28).

The interpolation processing unit 60 determines whether or not theinterpolation (generation of the pixel value) of all phase differencepixels (the first phase difference pixel PR and the second phasedifference pixel PL) in the imaging element 16 is finished (step S39).In a case where the interpolation of all phase difference pixels is notfinished (in the case of “No”), the interpolation processing unit 60returns to step S14, and the processes of step S14 to step S28 arerepeated. In a case where the interpolation of all phase differencepixels is finished (in the case of “Yes”), the process of theinterpolation processing unit 60 is finished.

According to the image processing method of the first embodiment, theaccuracy of the average value interpolation can be increased using theaddition pixel. In addition, the addition pixel is not used in theaverage value interpolation on condition that signal saturation causinga decrease in image quality is caused by using the addition pixel. Thus,the interpolation accuracy of the average value interpolation can bemaintained (image quality can be maintained).

FIG. 16 is a flowchart illustrating an image processing method accordingto a second embodiment of the present invention and particularly,illustrates a process procedure in the interpolation processing unit 60illustrated in FIG. 13.

In FIG. 16, common steps in the first embodiment illustrated in FIG. 15will be designated by the same step numbers, and detailed description ofsuch steps will not be repeated.

The image processing method of the second embodiment illustrated in FIG.16 is different from the first embodiment in that processes of step S40and step S42 are performed instead of the process of step S22 of thefirst embodiment illustrated in FIG. 15.

The flatness determination unit 68 illustrated in FIG. 13 calculates theflatness of the image in the predetermined window with the pixelposition of the in-focus pixel as a reference (step S40) and outputs thedetermination result indicating whether or not the image in the windowis flat using the calculated flatness to the average value interpolationunit 62 (step S40).

In a case where it is determined that the image surrounding the pixelposition of the in-focus pixel (in the predetermined window) is not flatbased on the determination result input from the flatness determinationunit 68 (in the case of “No”), the average value interpolation unit 62transitions to step S24. In a case where it is determined that the imageis flat (in the case of “Yes”), the average value interpolation unit 62transitions to step S26 (step S42). In step S24, the average valueinterpolation unit 62 performs the average value interpolation on thein-focus pixel using the normal pixels (G pixels) surrounding thein-focus pixel and also the addition pixel. In step S26, the averagevalue interpolation unit 62 performs the average value interpolation onthe in-focus pixel using only the normal pixels (G pixels) surroundingthe in-focus pixel without using the addition pixel.

According to the image processing method of the second embodiment, theaccuracy of the average value interpolation can be increased using theaddition pixel. In addition, the addition pixel is not used in theaverage value interpolation in the flat portion of the image whereerroneous correction is likely to stand out and causes a decrease inimage quality in a case where the erroneous correction (erroneousinterpolation) is performed using the addition pixel. Thus, theinterpolation accuracy (image quality) of the average valueinterpolation can be maintained.

<Second Embodiment of Imaging Element>

FIG. 17 is a diagram illustrating a second embodiment of color filterarrangement and arrangement of the phase difference detection pixels inthe imaging element 16.

The first periodic color arrangement of the color filters of the imagingelement 16 of the second embodiment is the general Bayer arrangement.

In the imaging element 16 having the Bayer arrangement, normal pixelrows in which only the normal pixels are arranged in the horizontaldirection (row direction) include an RG row in which a pixel (R pixel)having the R filter and a pixel (G pixel) having the G filter arealternately arranged in the row direction, and a GB row in which the Gpixel and a pixel (B pixel) having the B filter are alternately arrangedin the row direction. In addition, the RG row and the GB row arealternately arranged in the vertical direction (column direction).

In addition, the imaging element 16 of the second embodiment includesthe phase difference pixel row in which the first phase difference pixelPR and the second phase difference pixel PL are disposed, and the normalpixel row in which only the normal pixels are disposed.

The phase difference pixel row of the imaging element 16 illustrated inFIG. 17 is configured by periodically arranging three pixels includingthe pair of the first phase difference pixel PR and the second phasedifference pixel PL and one normal pixel as one cycle in the rowdirection in a specific GB row of the Bayer arrangement. Accordingly, inthe phase difference pixel row, the G pixel and the B pixel arealternately arranged in the row direction at an interval of two pixels(the pair of the first phase difference pixel PR and the second phasedifference pixel PL). While the phase difference pixel row of thepresent example is disposed in the GB row of the Bayer arrangement, thephase difference pixel row is not for limitation purposes and may bedisposed in the RG row.

In addition, for example, while the G filter is arranged in each of thefirst phase difference pixel PR and the second phase difference pixel PLof the present example, the G filter may not be arranged, and light in awavelength range wider than the transmission wavelength range of the Gfilter may be set to incident on the first phase difference pixel PR andthe second phase difference pixel PL.

In a case where the pixel values of the pair of the first phasedifference pixel PR and the second phase difference pixel PL in thephase difference pixel row are added, the added pixel value of thepixels (addition pixel value) is almost equal to the pixel value of thenormal pixel (G pixel). In addition, the added pixel (addition pixel)can be regarded as being present between the pair of the first phasedifference pixel PR and the second phase difference pixel PL.

The “average value interpolation” of the phase difference pixel can beperformed using a plurality of normal pixels present around the focusedphase difference detection pixel (the first phase difference pixel PR orthe second phase difference pixel PL) and the addition pixel in the samemanner as the imaging element 16 of the second embodiment having theX-Trans arrangement.

FIG. 18 is a diagram for describing the average value interpolation forthe phase difference detection pixel in the imaging element of thesecond embodiment.

The in-focus pixel which is the interpolation target illustrated in FIG.18 is the first phase difference pixel PR. The position of the in-focuspixel corresponds to the position of the G pixel.

In a case where the position of the in-focus pixel corresponds to theposition of the G pixel, the average value interpolation unit 62illustrated in FIG. 9 performs the interpolation operation on the pixelvalue of the in-focus pixel using the addition pixel value of theaddition pixel calculated by the pixel value addition unit 64 as thepixel value of one pixel of the pixels surrounding the in-focus pixel.

In a case where the signal gradient direction calculated by the signalgradient calculation unit 63 is the horizontal direction, the averagevalue interpolation unit 62 generates the pixel value at the pixelposition of the in-focus pixel by interpolating (calculating theweighted average depending on the distance) the addition pixel value ofthe addition pixel (two addition pixels illustrated by thick frames inFIG. 18) corresponding to the G pixel in the horizontal direction.

In addition, in a case where the signal gradient direction calculated bythe signal gradient calculation unit 63 is the vertical direction, theaverage value interpolation unit 62 generates the pixel value at thepixel position of the in-focus pixel by interpolating the pixel value ofthe G pixel in the vertical direction (two G pixels illustrated byarrows in the vertical direction).

Similarly, in a case where the signal gradient direction calculated bythe signal gradient calculation unit 63 is the +45 degree direction, theaverage value interpolation unit 62 generates the pixel value at thepixel position of the in-focus pixel by interpolating the G pixel in the+45 degree direction (two G pixels illustrated by arrows in the +45degree direction). In a case where the signal gradient direction is the−45 degree direction, the average value interpolation unit 62 generatesthe pixel value at the pixel position of the in-focus pixel byinterpolating the G pixel in the −45 degree direction (two G pixelsillustrated by arrows in the −45 degree direction).

In a case where the normal pixels surrounding the in-focus pixel aresaturated, or, for example, in the case of an image in which thesurrounding area of the in-focus pixel is even, it is preferable not touse the addition pixel value in the average value interpolation.

FIG. 19 is another diagram for describing the average valueinterpolation for the phase difference detection pixel in the imagingelement of the second embodiment.

The in-focus pixel which is the interpolation target illustrated in FIG.19 is the second phase difference pixel PL. The position of the in-focuspixel corresponds to the position of the B pixel.

In a case where the position of the in-focus pixel corresponds to theposition of the B pixel, the average value interpolation unit 62performs the interpolation operation on the pixel value of the in-focuspixel using only the B pixel surrounding the in-focus pixel withoutusing the addition pixel value of the addition pixel added by the pixelvalue addition unit 64. The reason is that the addition pixelcorresponds to the virtual G pixel.

In a case where the signal gradient direction calculated by the signalgradient calculation unit 63 is the horizontal direction, the averagevalue interpolation unit 62 generates the pixel value at the pixelposition of the in-focus pixel by interpolating (calculating theweighted average depending on the distance) the pixel value of the Bpixel in the horizontal direction (two B pixels illustrated by arrows inthe horizontal direction).

In addition, in a case where the signal gradient direction calculated bythe signal gradient calculation unit 63 is the vertical direction, theaverage value interpolation unit 62 generates the pixel value at thepixel position of the in-focus pixel by interpolating the pixel value ofthe B pixel in the vertical direction (two B pixels illustrated byarrows in the vertical direction).

Similarly, in a case where the signal gradient direction calculated bythe signal gradient calculation unit 63 is the +45 degree direction, theaverage value interpolation unit 62 generates the pixel value at thepixel position of the in-focus pixel by interpolating the B pixel in the+45 degree direction (two B pixels illustrated by arrows in the +45degree direction). In a case where the signal gradient direction is the−45 degree direction, the average value interpolation unit 62 generatesthe pixel value at the pixel position of the in-focus pixel byinterpolating the B pixel in the −45 degree direction (two B pixelsillustrated by arrows in the −45 degree direction).

The aspect of the imaging apparatus to which the present invention canbe applied is not limited to the imaging apparatus 10 illustrated inFIG. 1 and is exemplified by, for example, a mobile phone having acamera function, a smartphone, personal digital assistants (PDA), and aportable game console. Hereinafter, one example of the smartphone towhich the present invention can be applied will be described.

<Configuration of Smartphone>

FIG. 20 is a diagram illustrating an exterior of a smartphone as oneembodiment of the imaging apparatus.

A smartphone 100 illustrated in FIG. 20 includes a casing 102 having aflat plate shape. A display and input unit 120 in which a display panel121 as a display unit and an operation panel 122 as an input unit areformed as a single unit is disposed on one surface of the casing 102. Inaddition, the casing 102 comprises a speaker 131, a microphone 132, anoperation unit 140, and a camera unit 141 (imaging unit). Theconfiguration of the casing 102 is not for limitation purposes. Forexample, a configuration in which the display unit and the input unitare independently disposed can be employed, or a configuration having afolded structure or a sliding mechanism can be employed.

FIG. 21 is a block diagram illustrating an internal configuration of thesmartphone 100 illustrated in FIG. 20. As illustrated in FIG. 21, mainconstituents of the smartphone 100 comprise a wireless communicationunit 110, the display and input unit 120, a call unit 130, the operationunit 140, the camera unit 141, a storage unit 150, an externalinput-output unit 160 (output unit), a global positioning system (GPS)reception unit 170, a motion sensor unit 180, a power supply unit 190,and a main control unit 101. In addition, a main function of thesmartphone 100 includes a wireless communication function of performingmobile wireless communication with a base station apparatus through amobile communication network.

The wireless communication unit 110 performs wireless communication withthe base station apparatus connected to the mobile communication networkin accordance with an instruction from the main control unit 101. Byusing the wireless communication, transmission and reception of variousfile data such as voice data and image data, electronic mail data, andthe like and reception of web data, streaming data, and the like areperformed.

The display and input unit 120 is a so-called touch panel comprising theoperation panel 122 arranged on the screen of the display panel 121. Thedisplay and input unit 120 visually delivers information to the user bydisplaying images (still image and motion image), text information, andthe like and detects a user operation performed on the displayedinformation under control of the main control unit 101. The operationpanel 122 is referred to as a touch panel for convenience.

The display panel 121 uses a liquid crystal display (LCD), an organicelectro-luminescence display (OELD), or the like as a display device.The operation panel 122 is a device that is disposed in a state wherethe image displayed on the display surface of the display panel 121 canbe visually recognized, and detects one or a plurality of coordinatesoperated by a finger of the user or a stylus. In a case where the deviceis operated by the finger of the user or the stylus, the operation panel122 outputs a detection signal generated by the operation to the maincontrol unit 101. Next, the main control unit 101 detects the operationposition (coordinates) on the display panel 121 based on the receiveddetection signal.

The display panel 121 and the operation panel 122 of the smartphone 100illustrated in FIG. 20 constitute the display and input unit 120 as asingle unit. The operation panel 122 is arranged to completely cover thedisplay panel 121. In the case of employing such an arrangement, theoperation panel 122 may have a function of detecting the user operationeven in a region outside the display panel 121. In other words, theoperation panel 122 may comprise a detection region (hereinafter,referred to as a “display region”) for an overlapping part in overlapwith the display panel 121 and a display region (hereinafter, referredto as a “non-display region”) for the other peripheral part not inoverlap with the display panel 121.

The size of the display region may completely match the size of thedisplay panel 121. Both sizes do not necessarily match. In addition, theoperation panel 122 may comprise two sensitive regions including theperipheral part and the other inner part. Furthermore, the width of theperipheral part is appropriately designed depending on the size and thelike of the casing 102. Furthermore, a position detection methodemployed in the operation panel 122 is exemplified by a matrix switchmethod, a resistive film method, a surface acoustic wave method, aninfrared method, an electromagnetic induction method, an electrostaticcapacitive method, and the like. Any method may be employed.

The call unit 130 comprises the speaker 131 and the microphone 132. Thecall unit 130 converts the voice of the user input through themicrophone 132 into voice data processable in the main control unit 101and outputs the voice data to the main control unit 101, or decodes thevoice data received by the wireless communication unit 110 or theexternal input-output unit 160 and outputs the decoded voice data fromthe speaker 131. As illustrated in FIG. 20, for example, the speaker 131and the microphone 132 can be mounted on the same surface as the surfaceon which the display input unit 120 is disposed.

The operation unit 140 is a hardware key using a key switch or the likeand receives an instruction from the user. For example, as illustratedin FIG. 20, the operation unit 140 is a push-button type switch that ismounted on a side surface of the casing 102 of the smartphone 100. In acase where the operation unit 140 is pressed by the finger or the like,the operation unit 140 enters a switch ON state. In a case where thefinger is released, the operation unit 140 enters a switch OFF state bya restoring force of a spring or the like.

The storage unit 150 stores a control program and control data of themain control unit 101, address data in which a name, a telephone number,and the like of a communication counterpart are associated, data oftransmitted and received electronic mails, web data downloaded by webbrowsing, downloaded contents data, and the like and also temporarilystores streaming data and the like.

In addition, the storage unit 150 is configured with an internal storageunit 151 incorporated in the smartphone and an external storage unit 152including a slot for detachable external memory. Each of the internalstorage unit 151 and the external storage unit 152 constituting thestorage unit 150 is implemented using a storage medium such as a memoryof a flash memory type, a hard disk type, a multimedia card micro type,or a card type, a random access memory (RAM), or a read only memory(ROM).

The external input-output unit 160 acts as an interface for all externalapparatuses connected to the smartphone 100 and is directly orindirectly connected to other external apparatuses by communication andthe like (for example, Universal Serial Bus (USB) and IEEE 1394) ornetworks (for example, a wireless local area network (LAN), Bluetooth(registered trademark), radio frequency identification (RFID), infrareddata association (IrDA), Ultra Wideband (UWB) (registered trademark),and ZigBee (registered trademark)).

For example, the external apparatuses connected to the smartphone 100include a wired/wireless headset, a wired/wireless external charger, awired/wireless data port, a memory card or a subscriber identity module(SIM)/user identity module (UIM) card connected through a card socket,an external audio and video apparatus connected through an audio andvideo input/output (I/O), an external audio and video apparatusconnected in a wired/wireless manner, a smartphone, a personal computer,a personal digital assistant (PDA), and an earphone. The externalinput-output unit 160 may be configured to deliver data transferred fromthe external apparatuses to each constituent inside the smartphone 100or transfer data inside the smartphone 100 to the external apparatuses.

The GPS reception unit 170 receives GPS signals transmitted from GPSsatellites ST1, ST2 to STn, executes a position measurement calculationprocess based on the plurality of received GPS signals, and obtainspositional information (GPS information) specified by the latitude, thelongitude, and the altitude of the smartphone 100 in accordance with aninstruction from the main control unit 101. In a case where thepositional information can be obtained from the wireless communicationunit 110 and/or the external input-output unit 160 (for example, awireless LAN), the GPS reception unit 170 can detect the position usingthe positional information.

The motion sensor unit 180 comprises, for example, a three-axisacceleration sensor and detects a physical motion of the smartphone 100in accordance with an instruction from the main control unit 101. Bydetecting the physical motion of the smartphone 100, the movementdirection and the acceleration of the smartphone 100 are detected. Theresult of the detection is output to the main control unit 101.

The power supply unit 190 supplies power stored in a battery (notillustrated) to each unit of the smartphone 100 in accordance with aninstruction from the main control unit 101.

The main control unit 101 comprises a microprocessor, operates inaccordance with the control program and the control data stored in thestorage unit 150, and manages and controls each unit of the smartphone100. In addition, the main control unit 101 comprises a mobilecommunication control function of controlling each unit of acommunication system and an application processing function in order toperform voice communication and data communication through the wirelesscommunication unit 110.

The application processing function is implemented by operating the maincontrol unit 101 in accordance with application software stored in thestorage unit 150. For example, the application processing functionincludes an infrared communication function of performing datacommunication with an opposing apparatus by controlling the externalinput-output unit 160, an electronic mail function of transmitting andreceiving electronic mails, and a web browsing function of browsing webpages, and also includes an image processing function according to theembodiment of the present invention.

In addition, the main control unit 101 has the image processing functionsuch as displaying a video on the display and input unit 120 based onimage data (data of a still image or a motion image) such as receptiondata and downloaded streaming data. In addition, the image processingfunction includes image processing performed by the image processingunit 24 illustrated in FIG. 3.

Furthermore, the main control unit 101 executes display control of thedisplay panel 121 and operation detection control for detecting the useroperation performed through the operation unit 140 or the operationpanel 122.

By executing the display control, the main control unit 101 displays anicon for starting the application software or a software key such as ascroll bar, or displays a window for composing an electronic mail. Thescroll bar refers to a software key for receiving an instruction to movea display part of an image for a large image or the like notaccommodated in the display region of the display panel 121.

In addition, by executing the operation detection control, the maincontrol unit 101 detects the user operation performed through theoperation unit 140, receives an operation performed on the icon throughthe operation panel 122 or an input of a text string in an input fieldof the window, or receives a request for scrolling the display imagethrough the scroll bar.

Furthermore, by executing the operation detection control, the maincontrol unit 101 comprises a touch panel control function of determiningwhether the operation position on the operation panel 122 corresponds tothe overlapping part (display region) in overlap with the display panel121 or the other peripheral part (non-display region) not in overlapwith the display panel 121 and controlling the sensitive region of theoperation panel 122 and the display position of the software key.

In addition, the main control unit 101 can detect a gesture operationperformed on the operation panel 122 and execute a present functiondepending on the detected gesture operation. The gesture operation isnot a simple touch operation in the related art and means an operationof drawing a trajectory by the finger or the like, specifying aplurality of positions at the same time, or an operation of acombination thereof by drawing a trajectory from at least one of theplurality of positions.

The camera unit 141 converts the image data obtained by imaging intocompressed image data in, for example, joint photographic experts group(JPEG) and records the image data in the storage unit 150 or outputs theimage data through the external input-output unit 160 or the wirelesscommunication unit 110 under control of the main control unit 101. Asillustrated in FIG. 20, in the smartphone 100, the camera unit 141 ismounted on the same surface as the display and input unit 120. However,the mounting position of the camera unit 141 is not for limitationpurposes, and the camera unit 141 may not be mounted on the side surfaceof the casing 102 on which the display and input unit 120 is disposed.The camera unit 141 may be mounted on the rear surface of the casing102, or a plurality of camera units 141 may be mounted on the casing102. In a case where the plurality of camera units 141 are mounted,imaging may be performed by a single camera unit 141 by switching thecamera unit 141 performing the imaging, or imaging may be performedusing the plurality of camera units 141 at the same time.

In addition, the camera unit 141 can be used in various functions of thesmartphone 100. For example, the image obtained by the camera unit 141may be displayed on the display panel 121, or the image captured andobtained in the camera unit 141 may be used as one of operation inputmethods for the operation panel 122. In addition, in the detection ofthe position by the GPS reception unit 170, the position may be detectedwith reference to the image from the camera unit 141. Furthermore, withreference to the image from the camera unit 141, a determination of theoptical axis direction of the camera unit 141 of the smartphone 100 anda determination of the current usage environment can be performedwithout using the three-axis acceleration sensor or along with thethree-axis acceleration sensor. The image from the camera unit 141 canalso be used in the application software.

Besides, data obtained by adding the positional information obtained bythe GPS reception unit 170, voice information (may be text informationobtained by performing voice-to-text conversion by the main control unitor the like) obtained by the microphone 132, attitude informationobtained by the motion sensor unit 180, and the like to the image dataof the still picture or the motion picture can be recorded in thestorage unit 150 or output through the external input-output unit 160 orthe wireless communication unit 110.

[Others]

In the imaging element of the present embodiment, while the G filter isarranged in the phase difference detection pixel, the phase differencedetection pixel may be configured such that light in a wider wavelengthrange than the transmission wavelength range of the G filter can beincident on the phase difference detection pixel. For example, atransparent filter can be used without disposing the G filter in thephase difference detection pixel. Accordingly, a high pixel value can beobtained from even the phase difference detection pixel having a smalleropening portion than the normal pixel (the phase difference detectionpixel can have high sensitivity).

In addition, while the phase difference pixel row in each of the firstembodiment of the imaging element illustrated in FIG. 4 and the secondembodiment of the imaging element illustrated in FIG. 17 is configuredby periodically arranging three pixels including the pair of the firstphase difference pixel PR and the second phase difference pixel PL andone normal pixel as one cycle, the phase difference pixel row is not forlimitation purposes. For example, in the case of the first embodiment ofthe imaging element illustrated in FIG. 4, the phase difference pixelrow may be configured by periodically arranging six pixels including thepair of the first phase difference pixel PR and the second phasedifference pixel PL and four normal pixels as one cycle. In addition, inthe case of the second embodiment of the imaging element illustrated inFIG. 17, the phase difference pixel row may be configured byperiodically arranging five pixels including the pair of the first phasedifference pixel PR and the second phase difference pixel PL and threenormal pixels as one cycle or periodically arranging the first filter,the plurality of second filters respectively corresponding to two ormore colors other than the first color, and six pixels including thepair of the first phase difference pixel PR and the second phasedifference pixel PL and four normal pixels as one cycle.

Furthermore, the color filter arrangement is not limited to the X-Transarrangement illustrated in FIG. 4 and the Bayer arrangement illustratedin FIG. 17. The color filter arrangement may be any periodic arrangementof the first filter corresponding to the first color most contributingto obtaining the brightness signal and the plurality of second filtersrespectively corresponding to two or more colors other than the firstcolor.

In addition, in the present embodiment, for example, the hardwarestructures of processing units executing various processes like theimage processing unit 24 and the interpolation processing unit 60correspond to various processors as illustrated below. Variousprocessors include a central processing unit (CPU) that is ageneral-purpose processor functioning as various processing units byexecuting software (program), a programmable logic device (PLD) such asa field programmable gate array (FPGA) that is a processor capable ofchanging a circuit configuration after manufacturing, a dedicatedelectric circuit such as an application specific integrated circuit(ASIC) that is a processor having a circuit configuration dedicatedlydesigned to execute a specific process, and the like.

One processing unit may be configured with one of the various processorsor may be configured with two or more processors (for example, aplurality of FPGAs or a combination of a CPU and an FPGA) of the sametype or different types. In addition, a plurality of processing unitsmay be configured with one processor. A first example of configuring aplurality of processing units with one processor is such that oneprocessor is configured with a combination of one or more CPUs andsoftware, and the processor functions as a plurality of processingunits. A second example is such that a processor that implements thefunction of the whole system including the plurality of processing unitsusing one integrated circuit (IC) is used, as represented by a system onchip (SoC) or the like. Various processing units are configured usingone or more of the various processors as a hardware structure.Furthermore, the hardware structure of the various processors is morespecifically an electric circuit (circuitry) in which circuit elementssuch as semiconductor elements are combined.

In addition, the present invention is not limited to the aboveembodiments, and various modifications can be made without departingfrom the spirit of the present invention.

EXPLANATION OF REFERENCES

-   -   1: strobe    -   2: shutter button    -   3: power supply/mode switch    -   4: mode dial    -   5: zoom button    -   5T: tele button    -   5W: wide button    -   6: cross button    -   7: MENU/OK button    -   8: playback button    -   9: BACK button    -   10: imaging apparatus    -   12: imaging lens    -   14: stop    -   15: mechanical shutter    -   16: imaging element    -   16A, 16B: light shielding film    -   22: image input controller    -   24: image processing unit    -   26: compression/expansion processing unit    -   28: video encoder    -   30: liquid crystal monitor    -   32: sensor drive unit    -   33: shutter drive unit    -   34: stop drive unit    -   36: lens drive unit    -   38: operation unit    -   40: CPU    -   42: phase difference detection unit    -   44: AE detection unit    -   47: ROM    -   48: memory    -   50: VRAM    -   52: media controller    -   54: memory card    -   60: interpolation processing unit    -   61: gain interpolation unit    -   62: average value interpolation unit    -   63: signal gradient calculation unit    -   64: pixel value addition unit    -   65: final pixel value decision unit    -   66: addition pixel level correction unit    -   67: saturation determination unit    -   68: flatness determination unit    -   100: smartphone    -   101: main control unit    -   102: casing    -   110: wireless communication unit    -   120: display and input unit    -   121: display panel    -   122: operation panel    -   130: call unit    -   131: speaker    -   132: microphone    -   140: operation unit    -   141: camera unit    -   150: storage unit    -   151: internal storage unit    -   152: external storage unit    -   160: external input-output unit    -   170: GPS reception unit    -   180: motion sensor unit    -   190: power supply unit    -   CF: color filter    -   L: microlens    -   P: basic arrangement pattern    -   PD: photodiode    -   PR: first phase difference pixel    -   PL: second phase difference pixel

What is claimed is:
 1. An imaging apparatus comprising: an imager inwhich a plurality of phase difference detection pixels and a pluralityof normal pixels are two-dimensionally arranged in a first direction anda second direction orthogonal to the first direction, the phasedifference detection pixels include a first phase difference pixel and asecond phase difference pixel including opening portions for pupilseparation at different positions in the first direction, and theopening portions of the first phase difference pixel and the secondphase difference pixel are adjacently arranged; and processing circuitryconfigured to: generate an addition pixel value at a pixel positionbetween the first phase difference pixel and the second phase differencepixel by adding pixel values of the first phase difference pixel and thesecond phase difference pixel; and set the first phase difference pixelor the second phase difference pixel as an in-focus pixel and generate apixel value at a pixel position of the in-focus pixel by using pixelvalues of pixels surrounding the pixel position of the in-focus pixel,the addition pixel value that has been added being used as a pixel valueof one pixel of the surrounding pixels.
 2. The imaging apparatusaccording to claim 1, wherein: the processing circuitry is furtherconfigured to correct the addition pixel value by multiplying theaddition pixel value that has been added by a set level adjustmentcoefficient; and the processing circuitry uses the addition pixel valuethat has been corrected.
 3. The imaging apparatus according to claim 1,wherein: the processing circuitry is further configured to calculatesignal gradients of the pixels surrounding the pixel position of thein-focus pixel; and the processing circuitry performs an interpolationoperation on the pixel value at the pixel position of the in-focus pixelusing a pixel value of a pixel selected based on the calculated signalgradients out of the surrounding pixels.
 4. The imaging apparatusaccording to claim 1, wherein: the processing circuitry is furtherconfigured to set the first phase difference pixel or the second phasedifference pixel as the in-focus pixel and generate the pixel value atthe pixel position of the in-focus pixel by gain interpolation; and theprocessing circuitry generates the pixel value at the pixel position ofthe in-focus pixel by the gain interpolation based on the pixel value ofthe in-focus pixel and gain interpolation information set for the pixelposition of the in-focus pixel.
 5. The imaging apparatus according toclaim 4, wherein the processing circuitry is further configured todecide a final pixel value at the pixel position of the in-focus pixelby selecting any one pixel value of two pixel values including the pixelvalue that has been generated at the pixel position of the in-focuspixel and the pixel value that has been generated at the pixel positionof the in-focus pixel or generating a pixel value obtained by weightedaddition of the two pixel values.
 6. The imaging apparatus according toclaim 1, further comprising an imaging optical system that forms asubject image on a light-receiving surface of the imager, wherein theprocessing circuitry is further configured to detect a phase differencebetween a first pixel value obtained from the first phase differencepixel of the imager and a second pixel value obtained from the secondphase difference pixel; and control the imaging optical system based onthe phase difference that has been detected.
 7. The imaging apparatusaccording to claim 1, wherein in the plurality of normal pixels, a firstfilter corresponding to a first color most contributing to obtaining abrightness signal and a plurality of second filters respectivelycorresponding to two or more colors other than the first color arearranged in a first periodic color arrangement.
 8. The imaging apparatusaccording to claim 7, wherein the processing circuitry uses at least onepixel value of the addition pixel value or a pixel value of the normalpixel in which the first filter is arranged.
 9. The imaging apparatusaccording to claim 7, wherein in a case where the pixel position of thein-focus pixel corresponds to a pixel position of the normal pixel inwhich the second filter is arranged, the processing circuitry uses onlythe normal pixel in which the second filter is arranged.
 10. The imagingapparatus according to claim 7, wherein in each of the first phasedifference pixel and the second phase difference pixel, the first filteris arranged, or light in a wavelength range wider than a transmissionwavelength range of the first filter is incident.
 11. The imagingapparatus according to claim 10, wherein: the processing circuitry isfurther configured to determine saturation of at least one pixel of thein-focus pixel, the first phase difference pixel or the second phasedifference pixel adjacent to the in-focus pixel, or the normal pixelwhich is adjacent to the in-focus pixel and in which the first filter isarranged; and in a case where the processing circuitry determines thatthe pixel is saturated, the processing circuitry uses only the pixelvalue of the normal pixel in which the first filter is arranged.
 12. Theimaging apparatus according to claim 10, wherein: the first filter is agreen filter allowing transmission in a wavelength range of green, andthe plurality of second filters include a red filter allowingtransmission in a wavelength range of red and a blue filter allowingtransmission in a wavelength range of blue; the first periodic colorarrangement arranged in the two-dimensionally arranged plurality ofphase difference detection pixels and the plurality of normal pixels isconfigured by alternately arranging, in the first direction and thesecond direction, a first arrangement that corresponds to 3×3 pixels andin which the green filters are arranged at a center and four corners,the red filters are arranged in the first direction on both sides of thegreen filter at the center, and the blue filters are arranged in thesecond direction, and a second arrangement that corresponds to 3×3pixels and in which the green filters are arranged at a center and fourcorners, the blue filters are arranged in the first direction on bothsides of the green filter at the center, and the red filters arearranged in the second direction; and the imager includes a phasedifference pixel row in which the first phase difference pixel and thesecond phase difference pixel are arranged at positions adjacent to thegreen filter in the first direction, and a normal pixel row in whichonly the normal pixel is arranged in the first direction.
 13. Theimaging apparatus according to claim 10, wherein: the first periodiccolor arrangement arranged in the plurality of normal pixels is a Bayerarrangement; the imager includes a phase difference pixel row in whichthe first phase difference pixel, the second phase difference pixel, andthe normal pixel are arranged in the first direction, and a normal pixelrow in which only the normal pixel is arranged in the first direction;and in the phase difference pixel row, three pixels including the firstand second phase difference pixels and one normal pixel as one cycle areperiodically arranged, and a green filter allowing transmission in awavelength range of green is arranged in the first phase differencepixel and the second phase difference pixel.
 14. An image processingmethod for an imaging apparatus comprising an imager in which aplurality of phase difference detection pixels and a plurality of normalpixels are two-dimensionally arranged in a first direction and a seconddirection orthogonal to the first direction, the phase differencedetection pixels include a first phase difference pixel and a secondphase difference pixel including opening portions for pupil separationat different positions in the first direction, and the opening portionsof the first phase difference pixel and the second phase differencepixel are adjacently arranged, the method comprising: generating anaddition pixel value at a pixel position between the first phasedifference pixel and the second phase difference pixel by adding pixelvalues of the first phase difference pixel and the second phasedifference pixel; selecting the first phase difference pixel or thesecond phase difference pixel that is not processed as an in-focuspixel; and generating a pixel value at a pixel position of the selectedin-focus pixel by using pixel values of pixels surrounding the pixelposition of the in-focus pixel, the addition pixel value being used as apixel value of one pixel of the surrounding pixels.
 15. The imageprocessing method according to claim 14, further comprising correctingthe addition pixel value by multiplying the addition pixel value by aset level adjustment coefficient, wherein the addition pixel valuecorrected using the level adjustment coefficient is used when the pixelvalue at the pixel position of the selected in-focus pixel is generated.16. The image processing method according to claim 14, wherein: in theplurality of normal pixels, a first filter corresponding to a firstcolor most contributing to obtaining a brightness signal and a pluralityof second filters respectively corresponding to two or more colors otherthan the first color are arranged in a first periodic color arrangement;and at least one pixel value of the addition pixel value or a pixelvalue of the normal pixel in which the first filter is arranged is usedwhen the pixel value at the pixel position of the selected in-focuspixel is generated.
 17. The image processing method according to claim14, further comprising calculating signal gradients of the pixelssurrounding the pixel position of the in-focus pixel, wherein aninterpolation operation is performed on the pixel value at the pixelposition of the in-focus pixel using a pixel value of a pixel selectedbased on the calculated signal gradients out of the surrounding pixelswhen the pixel value at the pixel position of the selected in-focuspixel is generated.